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Abstract 


Humans are traditionally bad monitors , especially over long periods 
of time on reliable systems, and they are being called upon to do this 
more and more as systems become further automated. Because of this, 
there is a need to find a way to display the monitoring information to the 
human operator in such a way that he can notice pertinent deviations in 
a timely manner. One possible solution is to use polar-star displays that 
will show deviations from normal in a more salient manner. A polar-star 
display uses a polygon ’s vertices to report values. An important question 
arises, though, of how the vertices should move. This experiment 
investigated two particular issues of how the vertices should move: (1) 
whether the movement of the vertices should be continuous or discrete 
and (2) whether the parameters that made up each vertex should always 
move in one direction regardless of parameter sign or move in both 
directions indicating parameter sign. The results indicate that relative 
movement direction is best. Subjects performed better with this 
movement type and they subjectively preferred it to the absolute 
movement direction. As for movement type, no strong preferences were 
shown. 


Introduction 

Humans are traditionally bad monitors, 
especially over long periods of time on reliable 
systems (ref. 1), and they are being called upon 
to do this more and more as systems become 
further automated. Because of this, there is a 
need to find a way to display the monitoring 
information in such a way that the operator can 
notice pertinent deviations in a timely manner. 
Research has shown that an automated monitor 
can aid humans in recognizing and dealing with 
failures (refs. 2 and 3). One possible solution to 
this is to use polar-star displays (ref. 4) that will 
show deviations from normal in a more salient 
manner. A polar-star display uses a polygon’s 
vertices to report values. 

A polar-star display was chosen for its 
emergent features. Buttigieg and Sanderson 
reported that these “emergent features can be 
effectively used in displays to support global 
failure detection if they clearly carry information 
about important system states” (ref. 5). 
Furthermore, Cooper reported that the 
“identification of single lines in coherent, object- 
like contexts can be superior to identification of 
the lines when presented alone” (ref. 6). 


With a polar-star type of display, a different 
grouping of information may be beneficial rather 
than just one parameter per vertex, as is 
traditionally displayed. In the aviation domain, 
these groupings must reflect the parameters 
pilots feel are necessary to safely complete a 
flight (refs. 7 and 8). 

An important question arises, though, once 
the groupings are determined: How should the 
vertices move? Since there is no longer a one to 
one mapping of one parameter to one vertex, 
combination methods of several parameters to 
one vertex needs to be established. These 
combination methods will thus partially 
determine how the vertices of the polar star 
display move. However, since this movement is 
key to the operator monitoring and diagnosing 
mission health, operator preferences on 
movement were deemed important enough to 
resolve before various parameter combination 
methods were studied. 

Experiment Objectives 

This experiment investigated two particular 
issues with how the vertices of a polar-star 
display should move. The particular issues 
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studied were (1) whether the movement of the 
vertices should be continuous or discrete and (2) 
whether the parameters that made up each vertex 
should always move in one direction regardless 
of parameter sign or move in both directions 
indicating parameter sign. The tasks that 
addressed these issues were (1) a simple desktop 
simulation of a polar star and (2) a survey asking 
for preferences on movement. 

Experimental Variables 

The two primary experimental variables for 
this experiment were the (1) movement type and 
(2) movement direction of the vertices. Both of 
these variables were within subject. A between 
subject variable that was considered in the 
analysis was whether the subject was a 
certificated pilot. 

Movement Type 

The two movement types were selected for a 
couple of reasons. One school of thought was 
that continuous movement would aid in the 
gathering of trend information. On the other 
hand, there were thoughts that discrete 
movement would be more “eye catching.” 
Another aspect supporting discrete movement 
was that “the total movement required for 
motion detection increases with increased 
exposure duration” (ref. 9). Discrete movement 
was thought to be helpful in this case since, 
theoretically, pilots would be looking at this 
display often to check the general status of the 
aircraft and mission and a sudden jump would 
be more obvious than a slow deviation. 

Continuous. Continuous movement (c) was 
updated constantly. Occasionally, a small jump 
occurred due to the parameter combining 
method used to determine movement direction. 

Discrete. Discrete movement (d) had particular 
jump points. These jump points were (1) zero, 
(2) half-way between zero and the beginning of 
an alert range, (3) the beginning of an alert 
range, (4) half-way from the beginning of an 
alert range to the end of an alert range, and 


(5) end of an alert range (fig. 1). 

Movement Direction 

Movement direction was considered for a 
couple of reasons. It has been reported that 
“direction of movement makes little difference” 
in detection (ref. 10); but, this display is also 
supposed to aid in making a preliminary 
diagnosis. Therefore, movement direction was 
included to see if it did aid in realizing the 
nature of the non-normal situation. 

Absolute. The absolute method (a) of 
combining the parameter values at each vertex 
was basically the average of the absolute values 
of each parameter at a particular index (eq. 1). 
Since the average value was always positive, the 
vertices always moved out. 

let n = number of parameters at a vertex 

n . . 
if x t = warning => absolute = |x,| 

n i i 

else if x, = caution => absolute = |x ; | 

, , C 1 ) 

else if x ; = advisory =^> absolute = |x, | 

n 

ZN 

else absolute = — 

n 

In this experiment, color bands for each 
parameter signified the alert ranges (fig. 2). A 
red band indicated a warning. An amber band 
signified a caution. Finally, a cyan band 
signified an advisory. 

Relative. The relative method (r) of combining 
the parameters basically took the largest 
magnitude of the parameters at a particular 
vertex and then applied the sign of that value to 
the number (eq. 2). Therefore, the vertices could 
move both in and out. 

relative = sgn{y}* |y| (2) 

where 

let n = number of parameters at a vertex (3) 
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if x, = warning => y = x, 

i=i 

n 

else if x, = caution => y = x, 

i = 1 

n 

else if x j = advisory => y = x, 

1=1 

else y = max^^x^K |x n |) 

Pilot Status 

Pilot status was determined by whether the 
subject was a certificated pilot. Pilot status was 
considered because a previous experiment 
determined that there were differences between 
pilots and non-pilots in the way they handled 
non-normal system events (refs. 11 and 12). 

Experiment Design 

Subjects 

Eleven people participated in this experiment 
as subjects. Ten were male and one was female. 
The average age was 43.7 years old. Six of the 
subjects were certificated pilots. 

Polar-Star Display 

The polar-star display each subject saw is 
shown in figure 3. The dotted circle indicated 
the normal or expected value. The dotted circle 
was included because other research indicated 
that it was easier to detect abnormalities when a 
standard was provided (ref. 6). For this 
experiment, the expected value was zero. If a 
parameter reported by a particular vertex 
reached an alert range, the vertex number and a 
small dot placed at the vertex changed to the 
alert level color. Movement outside of the circle 
indicated positive or absolute values while 
movement to the interior of the circle indicated a 
negative value. 

Accuracy Question Screen 

After one of the vertices reached an alert 
level, subjects had to answer questions about the 
non-normal situation they just saw. These 
questions were (1) which vertex had the 


problem, (2) which parameter had the problem, 
(3) was the alert in the alert range above zero or 
below zero, and (4) what was the alert level (fig. 
4). In some instances, the subject would not be 
able to resolve which parameter had the problem 
and/or if the alert was in the alert range above or 
below zero. In these cases, the subject was 
instructed to answer “Could Not Tell.” If the 
subject forgot or just did not know the answer, 
he was instructed to answer with “Don’t Know.” 

Display Questionnaire 

After the subject completed all the runs for a 
particular display, he filled out a questionnaire 
asking how easy or difficult it was to use the 
display (appendix A). He also completed a 
NASA-TLX worksheet asking for his workload 
on using the display (appendix B and ref. 13). 

Final Questionnaire 

At the end of all the data runs, each subject 
rank ordered their preferences for the four 
displays they just saw (appendix C). An area 
was also provided for general comments. 

Procedure 

When a subject first arrived, he received a 
verbal briefing on the purpose of this experiment 
(appendix D). He was then shown the polar-star 
display and its characteristics were explained to 
him. The parameters that made up each vertex 
were described to him (fig. 2). The movement 
type of the vertices and the movement direction 
were also detailed. The questions he would be 
answering after each run were shown to him 
(fig. 4) plus the questionnaire and NASA-TLX 
were shown and explained to the subject 
(appendix A and B). 

After this initial briefing, the subject moved 
to the computer where he would be performing 
the experiment. The polar-star display was 
programmed on an Octane Silicon Graphics 
Computer using VAPS (ref. 14). The movement 
type and movement direction that he would see 
for the first display type were explained to him. 
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At this point the experiment began with three 
practice runs using the first display. These 
practice runs behaved exactly like the six data 
runs that followed the practice runs except that 
no data were recorded. 

During each run, only one parameter would 
reach an alert condition and once it did, it would 
remain in that alert condition. Once the subject 
saw this and was fairly sure he could answer 
questions about what he just saw, he used the 
mouse to click on the “STOP” button (fig. 3). 
The question screen then replaced the polar-star 
display screen (fig. 4). 

The subject could answer the questions in 
any order he wanted. He could also use and was 
encouraged to use the figure detailing the 
parameters that made up each vertex in order to 
help him answer the questions (fig. 2). The 
parameter sheet was provided for a couple of 
reasons. First, it was not expected that the 
subject had the time to memorize each 
parameter’s alert ranges. Second, the polar-star 
display is to be used with the available systems 
displays, which include parameter displays. 
Therefore, the parameter sheet acted as the 
actual parameter display. The subject’s answers 
were not recorded until he clicked on the 
“NEXT” button. Once he selected the “NEXT” 
button, the next practice or data run began. 

After six data runs with a particular display 
configuration were finished, a screen telling the 
subject to complete the display preferences 
questionnaire and the NASA-TLX appeared. It 
also told the subject the next display 
combination he would see. 

When the subject completed the two question 
sheets, the next display combination was 
described to him. He then clicked on the 
“NEXT” button to start the three practice runs 
with that display combination. 

At the end of all the data runs, each subject 
filled out the final questionnaire. 


Dependent Measures 

Several dependent measures were recorded. 
The objective data measures: time looking at the 
polar-star display and time to answer questions 
were recorded by the computer. Also recorded 
by the computer were the subject’s responses to 
the accuracy questions after each data run. The 
rankings that the subjects gave during the 
display questionnaire, NASA-TLX, and the final 
questionnaire were the subjective dependent 
measures. 

Hypothesis 

When considering the three experimental 
variables and the objectives of this study, the 
following were hypothesized. For the factor of 
movement type, subjects would prefer 
continuous movement but movement would not 
affect objective performance because subjects 
would be only attending to this display. This 
lack of secondary tasks precluded the need for 
increased motion to detect a change (ref. 9). For 
movement direction, subjects would perform 
better and prefer the relative condition because it 
would be easier to determine which parameter 
had the problem and whether the parameter 
value was high or low. The last experimental 
variable, whether the subject was a certificated 
pilot, would affect subject workload and the 
ability of the subjects to determine what the non- 
normal situation was (ref. 12). 

Data Analysis 

SPSS® was used for all data analyses (ref. 
15). The display factor was the combination of 
movement type and movement direction. 

Time and Subjective Questionnaires 

For the time subjects watched the display, the 
time it took them to answer questions, and the 
total combined time, data was analyzed using a 
repeated measures design with trial as the 
repeated measure. The independent variables 
were display and pilot status. 
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The subjective questionnaires were analyzed 
using a one-way ANOVA. This was possible 
since subjects were directed to mark anywhere 
on a continuous scale. Again, the independent 
variables were display and pilot status. 

Accuracy Questions 

The accuracy questions that subjects 
answered after each data run were analyzed. In 
particular, a Mann-Whitney U statistic was used 
for pilot status and a Kruskal- Wallis H statistic 
was used for the display factor. 

Results 

Display Effects 

Display was significant for the accuracy of 
determining if the non-normal parameter value 
was greater or less than zero (X 2 (3)=24.07, 
/?<0.01). As can be seen in figure 5, the 
parameter combination that included relative 
movement direction resulted in the most 
accuracy when determining if the non-normal 
parameter value was high or low. Subjects also 
reported that it was easier to determine if the 
parameter was high or low using the relative 
movement direction (Z=4.63, /K0.01) (fig. 5). 
In fact, a Tukey HSD post-hoc test indicated that 
relative movement direction grouped together 
and absolute movement direction was a separate 
group. 

Display was also significant for the 
subjective measure of how difficult it was to 
determine which parameter had the alert 
(F(3,36)=3.15, /?<0.04) but was not significant 
for the objective measure of how accurate 
subjects were in determining which parameter 
had the alert (X 2 (3)=1.52, p< 0.70). The 
subjective rating essentially indicated that the 
relative movement direction was preferred over 
the absolute movement direction (fig. 6). A 
Dunnetf s T3 post-hoc test was not significant 
but the trends indicated suggest the above 
groupings. 

When considering the display as a whole, 


there were significant differences for movement 
acceptability and the ease of determining the 
status as a whole. Subjects rated the relative 
movement type displays higher than the absolute 
movement type displays (F(3,36)=4.10, /?<0.02) 
for movement acceptability (fig. 7). The same 
also held true when determining the overall 
status of the display (F(3,36)=3.26, p<0.04) 
(fig. 7). Once again, relative movement ranked 
higher than absolute movement. Movement 
acceptability results also indicated a near- 
significant interaction of display by pilot status 
(F(3,36)=2.76, /?<0.06). As can be seen in table 
1, pilot ratings were much less extreme than the 
non-pilot ratings. 


Table 1 - Display by Pilot Status Ratings for 
Movement Acceptability 


Display 

| Pilot Status Ratings (0-1) | 

Non-Pilot 

Pilot 

c ontinuous-relative 

0.94 

0.75 

c ontinuous-ab solute 

0.50 

0.72 

discrete -relative 

0.89 

0.77 

discrete -ab solute 

0.60 

0.73 


Note: (Mow, l=high 


Whether the subject was a certificated pilot 
was only significant for how long they viewed 
the display. There were no significant results 
with pilot status for diagnosis accuracy although 
as seen in table 2, the same trends do seem to be 
present as reported in reference 12. 


Table 2 - Diagnosis Accuracy 



Pilot Status 


Non-Pilot 

Pilot 

Diagnosis Accuracy 
(%) 

86 

87 

Previous Experiment 
Diagnosis Accuracy 
(%) (ref. 12) 

34 

62 


Lastly, when rank ordering the displays, 
subjects preferred relative movement to absolute 
movement (F(3,36)=44.03, /?<0.01) (fig. 8). 
Although not significant, workload ratings also 
show the same trend (f(3,36)=0.54, p<0.70) 
(fig. 8). 
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Times 

The time that subjects viewed the polar-star 
display was significant (F(l,36)=9.09, /?<0.01). 
As seen in table 3, pilots viewed the display 
longer than non-pilots did. The time that 
subjects took to answer the questions was not 
significant (F(l,36)=0.16, /?<0.13) but here 
pilots answered the questions faster than non- 
pilots (table 3). 


Table 3 - Time Viewing Polar- Star Display and Time 
To Answer Questions 



Pilot Status Time (sec) 

Non-Pilot 

Pilot 

View Display 

11.72 

14.36 

Questions 

14.68 

12.54 


Discussion 

For determining how the vertices should 
move on a polar-star display, the results indicate 
that relative movement direction is best. 
Subjects performed better with this movement 
type and they subjectively preferred it to the 
absolute movement direction. As for movement 
type, no strong preferences were shown. 

Overall, the results indicate that relative 
movement direction is best with either 
continuous or discrete movement type. Since 
this display will be used for monitoring purposes 
and a reference normal will be shown at all 
times, the follow-on experiment to see whether 
the polar-star display aids in monitoring mission 
and aircraft health and in making a preliminary 
diagnosis will use continuous-relative 
movement. 

Conclusions 

Since humans are traditionally bad monitors, 
and since they are being called upon to perform 
this function more and more as systems become 
further automated, there is a need to find a way 
to display the monitoring information to the 
human operator in such a way that he can notice 
pertinent deviations in a timely manner. One 
possible solution to this problem is to use polar- 


star displays that will show deviations from 
normal in a more salient manner. 

In the experiment just described, subjects did 
best and preferred relative movement direction 
over absolute movement direction. Subjects did 
not show a strong preference for movement 
type. 

The next experiment will employ the polar- 
star display in a more full-mission simulation to 
see if the display does, in fact, aid in monitoring 
mission and aircraft health and in making a 
preliminary diagnosis. The polar-star vertices 
will show relative direction with continuous 
movement. But since the results from this 
experiment did not definitively find that 
continuous movement was best, subjects will be 
asked for their preferences after the simulation 
in the next experiment. Their reported 
preferences after using the display in a more 
realistic situation will be taken into 
consideration in the further refinement of the 
polar-star display used for these purposes. 
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beginning of alert range 

Figure 1 - Discrete Jump Points 
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Figure 3 - Polar-Star Display 
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QUESTIONS 


Which vertex had the alert? 



Which parameter had the alert? 


The parameter level was: 




Figure 4 - Question Screen 
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Continuous Continuous Discrete Discrete 

Relative Absolute Relative Absolute 


Display 

Figure 5 - Ability and Ease of Determining Whether 
Alert was a Above or Below Zero 



Continous Continous Discrete Discrete 

Relative Absolute Relative Absolute 


Display 

Figure 7 - Movement Preferences and Ease of 
Determining Status Ratings 
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Rating (0=low - 1=high) 


Final Rating (0=low - 1 = h i g h) 



Continuous Continuous Discrete Discrete 

Relative Absolute Relative Absolute 


Display 

Figure 6 - Accuracy of Determining Which Parameter 
Had an Alert 



Continous Continuous Discrete Discrete 

Relative Absolute Relative Absolute 


Display 


Figure 8 - Final and Workload Ratings 







Subj: 


Date: 


Appendix A — Display Questionnaire 

Directions: Please place a mark anywhere along the horizontal line of the rating scale when answering 
each question. 


For the this MSG display, 


a. determining which parameter had the 
problem was 


very 

difficult 


very 

easv 


b. determining if the non-normal parameter 

value was above or below normal was very 

difficult 


very 

easy 


c. determining the alert level of the 
non-normal parameter value w r as 


very 

difficult 


very 

easy 


d. the movement of the display was 


very 

unacceptable 


very 

acceptable 


e. determining the status of the 
parameters w r as 


very 

difficult 


very 

easy 


Comments: 


PreMSG Exp 
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Appendix C — Final Questionnaire 

Directions: Please place a mark anywhere along the horizontal line of the rating scale when answering 
the question. 


Rank order the displays on the scale below: 


W'orst best 

where 1 ^Continuous-Highest 

2 : ^ContinuouS"Summati on 
3=Discrete-Highest 
^Discrete- Summation 


Comments: 


Thank you 


PreMSG Exp 
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Appendix D — MSG Pre-Test Experiment 

(Directions / Script for Pi) 


Background 

® This experiment is looking at a new display for overall mission monitoring and preliminary 
diagnosis that combines several parameters. 

® The display is a polar star where each vertex represents a parameter or a combination of 
parameters. 

• Need to determine how the vertices should move. 

Purpose 

This experiment will compare 4 different combinations of ways to move the vertices. 

Procedure 

® This is a 1 hr experiment. 

® You will have several scenarios where one of the vertices will move out of the green normal 
range. 

• After you determine which parameter is not normal, you will have to answer some questions 
about the alert you saw. 

® After using a particular display, you will fill-out the NASA TLX and a questionnaire on your 

likes and dislikes of the display. 

» After the data runs, you will complete a final questionnaire asking you for your preferences on 
the displays you just saw. 

Displays 

• The general layout of the display is seen in figure 1 . The dotted circle indicates the normal 
expected value. Each vertex is a combination of 2 parameters (see figure 2). This sheet will 
be available to you during the data runs. 

® Each vertex will move either continuously or discretely, 
o Continuous movement is self-explanatory. 

o For discrete movement, each vertex will jump to either on the dotted line, ___ the distance 
to the bottom of an alert range, the bottom of an alert range, __ the distance into the alert 
range, or the top of the alert range. 

® The vertices will show either the highest absolute value of the lumped parameters or a 
summation of the parameters. 

MSG Lite 
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o For the highest absolute value, the vertex will display the actual value of the highest 
absolute value of the parameters associated with that vertex (/.e.,max(jx 1 | J |x ? |K |x o |)) 
unless a parameter is in an alert range. In that case, the value of the parameter in the 
highest alert range (warnings will be shown before cautions which will be shown before 
advisories) will be displayed. A value less than 0 will be in and a value greater than 0 will 
be out. 

o For summations, the vertex will display the value where x is the parameter and n 

n 

is the total number of parameters lumped into that vertex. However, if a parameter is in 
an alert range, the vertex text and dot will be the same color as the highest alert reached. 
Movement will always be out (values are always positive). 

® Therefore, the 4 display combinations are Continuous-Highest, Continuous-Summation, 
Discrete-Highest, and Discrete-Summation. 

Alerts 

® You will see 3 alert levels: advisories, which are cyan; cautions, which are amber; and 
warnings, which are red (see figure 3). 

® You will only see 1 alert per data run. 

Questions After Each Data Run 

® After each data run, you will answer some questions about what just happened, 

® The questions are which vertex was not normal, which parameter was in the alert range, 
whether the alert was a high or low alert, and the alert level (advisory, caution, or warning) 
(see figure 4). 

® In some instances, you may not be able to answer a question because there is not enough 
information. In that case, please hit Could Not Tell. 

® If you do not know an answer to a particular question, hit Don t Know. 

NASA TLX Workload Rating Form 

® After completing all the data runs for a particular display combination, you will fill-out a NASA- 
TLX workload rating form and Display Questionnaire sheet (see sheets 1 and 2). 

Runs 

® There are 24 data runs. 

® Before the data runs, you will be given 3 practice runs that behave similarly to the data runs 
(see figure 5). 
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*> After the practice runs for a particular display, you will have 6 data runs (see figure 1). 

*> Once you know what parameter is out of bounds, hit the Stop button (see figure 1 ) to 
answer the questions (see figure 4). 

*> Once you have answered the questions for that particular data run, hit the Next button to go 
on to the next trial (see figures 4 and 1). 

* After the 6 data runs for a particular display, you will fill out the NASA — TLX and Display 
Preferences sheets (see sheets 1 and 2 and figure 8). 

* You will then have 3 more practice runs with the new display before continuing on to the data 
runs. 

® At the end, you will fill out 1 last question about your display preferences (see sheet 3). 
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